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ABSTRACT 



This paper discusses methodological difficulties in studying 
metacognition and proposes a new multi -method technique for assessing student 
metacognitive behavior. The major aim of the study was to develop and apply a 
strategy for assessing metacognition within the context of mathematical 
problem solving. Data were collected from 30 year-6 students in Australia. 
Students were involved in three clinical interviews each in which they 
followed the same procedure with different types of mathematical tasks. The 
multi-method approach included observation, the clinical interview, and video 
and audio recordings. The interviews incorporated self-reporting and, in some 
cases, the think-aloud technique. Specially designed metacognitive and 
cognitive action cards were used to stimulate student responses about their 
thinking. Findings raise dovibts about using students' "out of context" verbal 
reports as data without corroborating evidence. The findings also call into 
question the validity of questionnaires or interviews when used in isolation. 
Results also demonstrate the difficulty students have in identifying their 
general use of metacognitive behavior. The multi -method approach used in this 
study responded to issues of concern about verbal data and provided students 
with opportunities to identify, check, and discuss their thinking. (Contains 
37 references.) (SLD) 
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Metacognition has been identified as having an important role in many 
aspects of education, yet specific details about how students use 
metacognition when they tackle problems is not well researched. Research 
in the field has spanned three decades, therefore this field would not be 
considered a new frontier but the assessment of metacognition is. 

One of the difficulties associated with researching metacognition is that the 
term lacks clarity. The research implication is that parametres need to be 
defined to conduct assessment research. Another problem for researchers 
interested in metacognition is that techniques used in the research of 
metacognition, such as, verbal self-reporting are often criticised. 

This paper will discuss important methodological difficulties of 
researching metacognition and propose a viable, new multi-method 
technique used for assessing student metacognitive behavior. This new 
multi-method interview technique was used with Year 6 students from 
three different Victorian schools undertaking a range of non-routine 
mathematical problems but the results may well be applicable across other 
age groups and across different curriculum domains. 



Verbal reports can provide researchers with ambiguous and contradictory 
data about students' actual practices, but researchers in the field of cognitive 
study need self-reports as a source of data. The following student quotes 
from this research highlights the difficulties fpr researchers when trying to 
interpret meaning from student reports. 

"Well I always do that but most of the time I don't." (Louisa) 

"I never, hardly ever do it in the classroom." (Therese) 

"When I finish a problem I do check it, but when I don't ... 

(Peter) - 
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These student statements were made by during the process. As they stand, they 
provide ambiguous and doubtful data about students' actual practices and 
their thinking. There are a number of possible explanations for such data, for 
example students may be unsure about what they do, unable to express what 
they do and think, they may be inventing answers or trying to please the 
researcher. When Julie was asked to explain discrepencies in verbal data she 
responded: "Maybe I was off with the fairies or pixies or something?" (Julie) 

This article reports on verbal reports made by Year six students and the 
credibility of these in the context of a multi-method approach to researching 
metacognitive thinking. This paper raises important questions about students' 
verbal reports in the area of metacognition and discusses how the limitations 
of self-reports can be reduced. 



Aims and Terms 

The.major aim of this study was to develop and apply a strategy for assessing 
metacognition within the context of mathematical problem solving. As well as 
investigating the effectiveness of particular assessment strategies, the study 
enabled an examination of the nature of metacognition and the relationship 
between metacognition and task type. The focus of this paper is on 
methodological difficulties of assessing metacognition. 

One of the initial challenges for researchers in this field is defining the term 
metacognition. Metacognition was used in this research to refer to the 
awareness individuals have of their own thinking and their ability to 
evaluate and regulate their own thinking. The three functions of 
metacognition were identified as: Metacognitive Awareness, Metacognitive 
Evaluation and Metacognitive Regulation. 

Contextualising the Research 

The data collection involved thirty Year Six students in three different schools. 
Ninety interviews were conducted, with each student being interviewed three 
times. Students were involved in three clinical interviews (plus a 
familiarisation task) in which they followed the same procedure with different 
types of mathematical tasks. The multi-method approach included: 
observation; a problem-based clinical interview (incorporating self-reporting 
and in some cases the think aloud technique); video and audio recordings 

The most unique and revealing component of the clinical interview was the 
use especially designed metacognitive and cognitive action cards (see 
Appendix 1) to stimulate student responses about their thinking. This 
procedure is discussed in length later in this paper. 



While the centrality of metacognition in the problem solving process is widely 
recognised (Stacey 1990; Munro 1993), details of metacognitive behavior 
involved during problem solving are few (Goos, 1998). This is largely due to 
methodological problems (Garofalo and Lester 1985; Nisbett and Wilson 1977; 
Nuthall and Alton-Lee 1995). The multi-method interview was designed to 
document the nature of metacognition during mathematics problem solving 
and this paper reports on a compelling methodological findings. 

In the process of developing a new multi-method approach which could 
confidently be used with young students for studying metacognition, a 
number of checking procedures were included in the interview protocol 
(discussed later). One such procedure allowed the comparison of students’ 
comments made about metacognitive use within the context of particular 
problems, and student comments about their reported use of metacognition at 
other times. The results were surprising. These are discussed after a brief 
review of the literature on methodology and an overview of the Multi- 
method technique developed for this study. 

Verbal Self-Reporting Issues 

The use of verbal methods in the collection of data on cognitive and 
metacognitive processes is controversial (Ericsson and Simon 1980; Nisbett 
and Wilson, 1977; Nuthall and Alton-Lee, 1995). There are several 
questions related to the value of self-reporting and criticisms about the 
possible impact of self-reporting on cognitive processes. One of the major 
criticisms of verbal reports as a data collection devise is that specific 
thoughts, triggered by the interview process itself, may be reactive and may 
alter the cognitive thoughts being studied (Cavanaugh and Perlmutter 
1982). ‘ 

Meichenbaum, Burland, Gruson and Cameron, (1985) claimed, that the 
most recurrent and serious concerns relate to the accessibility-, veridicality 
and completeness of verbal reports. Such verbal report problems are even 
more an issue when dealing with children with limited linguistic 
capabilities. Cavanaugh and Perlmutter (1982) and Wellman (1985) are 
most critical of the verbal interview, claiming that it has the most 
'methodological liabilities and poorest track record' (1985 : 198) for 
assessing cognition, especially with children. 

Nisbett and Wilson (1977) seminal paper argued that asking students to 
generate a verbal report on a general procedure or strategy may not 
accurately describe their actual cognitive processes. But Ericsson and 



Simon (1980) analysed Nisbett and Wilson's (1977) work and found it to 
have limitations. Ericsson and Simon (1980) criticised the earlier work on 
several grounds including: an undefined framework and ignoring 
experimental conditions which may impact upon the results. They 
concluded that verbal reports can provide a valuable and reliable source of 
information about cognitive processes. 

Cognitive research focuses on behaviors which are not directly observable, 
therefore, indirect forms of measurement must be developed (Weinstein 
and Meyer, 1991). Variations of self-reporting have been developed in 
response to these concerns. Self-reporting is often associated with studies 
of metacognition. When a student is asked to self-report they may be given 
a hypothetical or real problem to discuss. They are asked to either 
concurrently, retrospectively or hypothetically explain the strategies they 
might apply or have applied to the problem. The reports are interpreted by 
the researcher as evidence of metacognition. This technique has 
sometimes involved thinking aloud. The think aloud technique requires 
students to verbally express their thoughts as they perform a task. 

Brown (1987) and Meichenbaum et al (1985) expressed wariness about the 
veridicality of self-reports. Brown alluded to the possible effects of self- 
reporting on the thinking processes: 'Asking subjects to report on internal 
events that are not readily available to such inspection may significantly 
impair the processes on which they must report.' (Brown, 1987 : 76). 

The timing of self-reporting has been raised by some researchers as an 
issue. This can be particularly problematic in studies of metacognition. The 
question of the accuracy of student post performance self-reports poses a 
dilemma for researchers is deciding whether reported metacognition 
represents what has occurred or what students think may have occurred 
(Meichenbaum, et al. 1985). Concurrent verbalisation technique with no 
interviewer intervention is least prone to study environment effects and 
to 'incompleteness and inconsistency of some verbal data' according to 
Scott (1994: 537) and that this is the most suitable for collecting problem 
solving data. 
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The most difficult problem when researching cognition, is when the 
mental process under question is one the subject does not normally pay 
attention to. Feil and Gatti (1993) stated that: 'Under these circumstances, 
obtaining a clear response or one that truly reflects the subject's actual 
internal processing may be difficult' (1993 : 630) but they suggested that one 
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method for overcoming these problems is to use specific 'thinking aloud' 
probes. 

Students may not be able to recall their metacognitive acts because they 
some of them may have become automatised (Ericsson and Simon, 1980; 
Me Koon and Ratcliff 1992). If students do not report cognitions it is 
difficult to determine whether the absence is actual or just not reported. 
Schoenfeld (1985) questioned whether verbal reports are accurate 
reflections of the processes used. Even with appropriate methodological 
care, he suggested that caution needs to be taken when interpreting verbal 
data. Schoenfeld (1985) argued that any methodology, can illuminate some 
aspects of behavior but distort others. He suggested that environmental 
factors, such as the perceived need of subjects to perform, their beliefs 
about the research requirements or their beliefs about the discipline can 
affect the results. Schoenfeld (1985) contended that verbal data should be 
compared and contrasted with data from other sources. This contention is 
well supported by others and basic to the technique developed in this 
study. 

To maximise the completeness of verbal reports, enquiry immediately 
after the event is preferable, the task should be short and probing should 
be minimised. Ericsson and Simon (1980) recommended that internal 
consistency of results must be examined. Further they claimed that in 
trying to say what one was thinking, the subject might not remember, 
might misremember, or invent memories (for example, describing 
strategies that have just occurred to them). Genest and Turk (1981) agreed 
that reports are likely to be incomplete. While Ginsburg, Kossan Schwartz 
and Swanson (1983) also argued that self-reports can never be complete, 
they add that no data source can ever be complete or provide aill answers 
needed. 

The literature identifies a number of problems, associated with verbal 
reports but any study which examines thinking processes must involve 
subjects reporting on their own thinking, therefore ways to minimise the 
limitations of verbal self-reporting must be investigated. 

Reducing the Limitations of Verbal Self-Reporting 
There are a range of suggestions made about ways to reduce the limitations 
of verbal reports. Some have been reviewed in the preceding section. 

Those discussed in this section are particularly pertinent to this research 
that is concerned about thinking processes and problem solving. Ericsson 
and Simon (1980) claimed that there are several issues that must be 



attended to when considering verbal reports as data. These include: the 
effects on the cognitive processes, the completeness of reports and the 
consistency of reports. Further they suggested that genera Usability and 
validity and the design of objective methods for encoding and analysing 
protocols are a concern. Ericsson and Simon (1980) argued that verbal 
reports should be collected with other records of behavior, consistency of 
results can then be checked. 

In order to obtain reliable verbal data, retrospective self-reports should be 
obtained as soon as possible after the event (Fell and Gatti 1993 and 
Meichenbaum, et al 1985) . Otherwise the accuracy of verbal reports 
deteriorates as the time between the 'event' and the interview increases. 
'Stimulated recall' (Miles and Huberman 1984) has often been used to 
gain access to the perceptions of the participants after the activity and 
without interrupting 'thought in action'. Usnick and Brown (1992) 
promoted the use of 'stimulated recall' as an effective tool to probe 
students problem solving thinking. By replaying recorded (audio or video) 
material and presenting work samples to students, retrieval cues (Adair 
and Spinner, 1979 and Meichenbaum et al, 1985) are provided to the 
student so that s/he can more easily explain what s/he was thinking at the 
time. 

Such cues have been promoted by Randhawa (1994), to capture the 
cognition of the problem solver. She discussed the difficulties associated 
with assessing internal actions: 'Problem solving in any domain requires, 
among other things, encoding and translation, which are internal and 
unobservable. To understand the process of problem solving these 
internal mechanisms must be externalised.' (1994 : 218). Thus Randhawa 
suggested videorecording of think-aloud protocols along with clinical 
interviews is non-intrusive. 

Adair and Spirmer (1979) also provided guidelines about enhancing the 
accessibility and veridicality of self-reports. They suggested that the 
researcher discusses the importance of honesty with subjects. The selection 
of tasks which legitimise disclosure of cognitions was also considered 
crucial by Adair and Spinner (1979). 

There seems to be no single verbal assessment method which cannot be 
criticised on some grounds. But Hacker, Dunlosky and Graesser, (1998) 
posited that if metacognition is defined as conscious and deliberate 
thoughts about one's own thinking, then these thoughts are potentially 
controllable, reportable and therefore accessible to the researcher. It is 



argued that a technique would be considered more reliable if it combined 
methods. The multi-method approach was designed to capitalise on the 
strengths of particular techniques and avoid the disadvantages of each 
individual method. 

Development of the Multi-Method Interview 
This study employed a new technique that drew on and combined the 
strengths of the many commonly used strategies. The multi-method interview 
was designed as a result of unsatifactory pilot study findings. The technique 
also responded to published criticisms of previous metacognitive research 
(Clarke, Stephens and Waywood, 1992; Clarke, 1996; Garofalo and Lester, 1985; 
Ginsburg et al, 1983), particularly those about verbal methods (Nisbett and 
Wilson, 1977 and Nuthall and Alton-Lee, 1995) and to the recommendation 
for new research methods to assess metacognition (Cavanaugh and Perlmutter 
1982; Dunlosky 1998; Garofalo and Lester, 1985; Meichenbaum et al, 1985; 

. Mulcahy, Short and Andrews, 1991; Randhawa, 1994). 

The new approach included: observation; a problem-based clinical interview 
(incorporating self-reporting, sometimes 'think aloud' and a card sorting task); 
video and audio recordings. The card sorting task drew upon items from an 
earlier pilot questionnaire (related to metacognitive and cognitive actions), a 
Likert type procedure was implemented in a more 'active hands on' way. A 
total of fourteen metacognitive action statements (called action cards), each 
associated with one of the three metacognitive functions (Awareness, 
Evaluation and Regulation), were listed individually on playing cards (see 
Appendix 1). Multiple statements were used to improve the validity of the 
measurement technique (Fowler 1984; Reid 1990). Cards listing cognitive 
behaviors and blank cards were also provided. 

During the clinical interviews students were asked to attempt a given 
mathematics problem. Students were familiar yv^ith the language on the 
cards and the card sorting routine. Their attempts were video- recorded. 

After attempting the given task students were asked to sort the cards into 
two piles: cards that represented their thinking during their problem 
solving and those that were not applicable. Students were then asked to 
place the cards in sequence according to how they solved the problem. 

Cards were provided to give students a means to express their thinking. 

The use of the action cards eliminated the "Don't know' responses which 
were given at the trialling stage when only questionnaires and structured 
interviews were used to assess student metacognition. Haynes (1997) 
supported the objective of providing cues for discussing thinking: 'How 



can one be metacognitively aware or reflective without a language which 
to think about oneself?' (1997: 6). 

Checking Procedures 

To ensure that the students' card sequence was an accurate representation 
of their thinking, after the students had placed the cards in order, the 
video of their problem solving attempt was replayed. While watching the 
video, students were asked to point to the particular card that represented 
their thinking and behavior at each moment of the problem solving 
process. The video was stopped as many times as the students wanted. If 
they wished to, students could change the cards by discarding, adding or 
changing the sequence. 

Students took the task of card sorting and checking very seriously, they took 
their time and often changed their selection when they rechecked the discard 
pile. The most frequent response was to add cards.. Students also sometimes 
changed the order and selection (more rarely) after watching the video. On 
most occasions, students changed their card sequence slightly (up to 4 changes). 
They usually added some cards by writing on the blank cards. Sometimes 
students eliminated some cards when checking their sequence against the 
video. There were some occasions when card sequences were not changed but 
every student changed their sequence at least once during the three 
interviews. 

At the end of session two, after completing three tasks (including the 
familiarisation task) students were given the metacognitive action 
statements only and asked to indicate if and how often they generally did 
these things when they solved mathematics problems. Students sorted 
these cards into self-chosen categories such as: always, usually, sometimes 
do and never. This task was used to check what students said they do 
when they solve mathematics problems against their reported actions 
during the interviews. In most cases this demonstrated that student 
reports of what they did during the multi-method interviews was not the 
same as what they said they generally do when they solve mathematics 
problems. Sometimes students said they never did a particular action, but 
they had reported this action in one or both of the multi-method 
interview/ s, or they said they always did a particular action but they had 
not reported doing it during either of the multi-method interviews. 

When discrepancies occurred students were asked during the following week 
(session three) to explain why they thought there may be a difference. It is 



noted that there was the only student who had used every metacognitive 
action statement during both interviews as she had claimed she generally did. 
This checking procedure raised questions about the difference between the 
student reports. 

Verbal Reporting as Data: In Context and in General 
The issue of contextual and temporal variation in metacognitive activity is 
highlighted by students' comments recorded during the second multi-method 
interview. Metacognition is dynamic, it responds to contextual and task 
features. Metacognition is activated according to challenge (such as in the non- 
routine tasks used in the study), when difficulties are encountered or when 
routines are not working. If everyday classroom tasks differ from the research 
tasks it would be expected that students would find it difficult to compare the 
way that they generally solve problems and how they relate their actions to 
those used during the multi-method interviews. 

A number of other reasons could be forwarded to help explain the question of 
the difference between the students' reports within context (of the multi- 
method interview) and generally (applying to mathematics problem solving 
in general): 

• Student reports of their own general behayior patterns are not accurate. 

• Many or most students are not aware of what they are doing. 

• Cards may be misinterpreted or interpreted in different ways on different 
occasions by individual students. 

• It is difficult for students to generalise out of context. 

• Different types of tasks require different types of responses, for example, 
students would not need to change the way they were working if their 
strategies were being successful. 

• The level of difficulty determines the need for. “particular cards and 

therefore the reported sequence, for example, if the task was not 
recognised as similar to one they have dope they may need to make a 
plan to work it out. . . ... 

• The multi-method interviews create a context seen by students as different 
to problem solving at other times. 

Although the first three possible reasons seem feasible these are not supported 
by the data. Student confidence in card selection and consistency in sequences 
across individuals, grades and schools suggests students can report their 
metacognitive thinking with the use of the action cards. 

An important methodological problem for many researchers is whether their 
questions or tasks (interview, questionnaire or in this case action cards) assess 



what the researcher wants to know (Cavanaugh and Perlmutter 1982). In order 
to check internal validity, one third of the students were asked to explain what 
they would be thinking if they selected particular action cards. Their responses 
confirmed that their interpretations were consistent with the intensions of the 
researcher for each card and demonstrated high internal validity for each card. 
Therefore misunderstandings of the cards was dismissed as a possible reason 
for discrepancies in reports. Student explanations for the differences in their 
reports mostly related to task variation, task difficulty and memory. 

As well as providing data on verbal reports, other conversations during the 
multi-method interviews yielded interesting data about the nature of 
metacognition and classroom activities but as these are not central to the topic 
of this paper they are not discussed here (see Wilson, 2000) 

Discussion and Conclusion 

This paper has addressed important methodological questions about using 
student's verbal reports of their own thinking as data. Of central interest to 
this report is the confidence which can be attached to what student's report 
about their metacognitive thinking. The results raise doubts and questions 
about using students 'out of context' verbal reports without as data without 
corroborating evidence. 

The results call into question the validity of questionnaires or interviews 
when used in isolation. The accuracy of either verbal data collection method, 
when used alone, as a suitable tool to measure a child's metacognitive 
repertoire must be questioned. Students statements (such as those included in 
this article) could not be easily interpreted without further clarification and in 
the light of other data. When verbal data is compared and contrasted against 
data from other sources, consistency of results can then be checked (Ericsson 
and Simon, 1980). Cohen and Manion (1994) , Schoenfeld (1985) and Wellman 
(1985) suggested the idea that one way to validate data is to compare it with 
another measure that is valid. If the two measures agree validity is assumed. 

Although Savage claimed that; 'Asking a child how he went about solving a 
problem is the next best thing to getting inside his or her head.' (Savage 1994), 
a notion supported by Ginsburg (1983) and Pines, Novak, Posner and Van 
Kirks, (1978), individual verbal reporting techniques do not provide hard 
evidence which may be documented with the additional use of cards and 
videorecordings that can be checked, rechecked, verified and justified. 

The results of this study also indicate the difficulty students have with 
identifying their general use of metacognitive behavior. It is asserted that 



verbal responses about metacognition must be considered in a practical context 
and examined immediately after the event. 

Any study involving children and the examination of their thinking processes 
must be involve self-reporting. While the limitations of self-reporting 
procedures are acknowledged, it is noted that none of the alternative 
assessments have proven to be unproblematic. Because of the influence of 
cognition and metacognition on behavior and learning, self-reporting 
methods must be developed and refined (Genest and Turk 1981) . 

It is suggested that the multi-method interview satisfactorily responded to 
issues of concern about verbal data as presented throughout this article. 
Consistency within student responses during the multi-method interviews is 
taken as indicative of the reliability of the multi-method interview for 
accessing student's metacognition during mathematical problem solving. This 
study has focussed on the assessment of metacognition within mathematics, 
but it could be adapted to other curriculum domains and for other age groups. 



The multi-method interview provides opportunities for students to identify, 
check and discuss their thinking. Instead of self-reports being considered being: 

"Off with the fairies or pixies or something." (Julie), verbal reports should 
then be interpreted, presented and viewed with more confidence. 
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Appendix 1 Action Statements 



The metacognitive action cards included: 

I thought about what I already know (Awareness) 

I tried to remember if I had ever done a problem like this before . (Awareness) 
I thought about something I had done another time that had been helpful 
(Awareness) 

I thought 7 know this sort of problem' (Awareness) 

I thought 7 know what to do' (Awareness) 

I thought 'Is this right?' (Evaluation) 

I thought 7 cant do it. ' (Evaluation) 

I thought about how I was going (Evaluation) 

I thought about whether what I was doing was working (Evaluation) 

I checked my answer as I was working (Evaluation) 

I made a plan to work it out (Regulation) 

I thought about a different way to solve the problem (Regulation) 

I thought about what I would do next (Regulation) 

I changed the way I was working (Regulation) 

The cognitive action cards for the logic and number tasks included: 

I asked for help 
I drew a diagram 
I read the question again 
I added 
I subtracted 
I multiplied 
I divided 
I counted 

The cognitive action cards for the tangram task included: 

I tried to see if a shape would fit 
I moved a shape around 
I turned a shape over 
I tried a different shape 
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